<html>
<head><meta charset="utf-8"><title>Decentralize queries compiler-team#277 · t-compiler/major changes · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/index.html">t-compiler/major changes</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html">Decentralize queries compiler-team#277</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="194545129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/194545129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> triagebot <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#194545129">(Apr 18 2020 at 13:39)</a>:</h4>
<p>A new proposal has been announced <a href="https://github.com/rust-lang/compiler-team/issues/277" title="https://github.com/rust-lang/compiler-team/issues/277">#277</a>. It will be brought up at the next meeting.</p>



<a name="194550646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/194550646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#194550646">(Apr 18 2020 at 15:40)</a>:</h4>
<p>The state of my understanding is:<br>
The current monolithic query plumbing does not need to be in librustc_middle.<br>
The list of all queries and their key types is required to define the DepKinds and DepNodes.<br>
I don't know how to remove the use of the DepGraph by TyCtxt.</p>



<a name="196290289"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/196290289" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#196290289">(May 05 2020 at 10:27)</a>:</h4>
<p>I think <span class="user-mention" data-user-id="248906">@cjgillot</span> we might want a bit more detail here in terms of the proposed approach. =) <span class="user-mention silent" data-user-id="119009">eddyb</span> described <a href="https://github.com/rust-lang/rust/pull/70951#issuecomment-612156842" title="https://github.com/rust-lang/rust/pull/70951#issuecomment-612156842">their preferred approach</a>. I don't have a strong opinion yet but I am in favor of the overarching goal. I'd actually like to see us reconciling a bit with salsa, longer term, but I'm not pushing on that yet, in part because I don't have enough energy to keep up with salsa anyway :P</p>



<a name="196345475"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/196345475" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#196345475">(May 05 2020 at 18:18)</a>:</h4>
<p>I don't really have a converged design in mind. I would like to get to something like @eddyb described. Each crate would be able to only define a set of queries. The same crate, or a downstream crate would then be able to <code>provide</code> it. </p>
<p>However, I don't yet have a plan to get there. AFAIK, the main blocker is the definition of the <code>DepKind</code> type, and how to get rid of it / render it opaque.<br>
From my experiments, all the query invocation system can be moved out of librustc_middle. I think the definition of the <code>Providers</code> struct would be movable. The issue is the <code>DepKind</code> enum, and the types in <code>rustc_middle::dep_graph</code>. They are intertwined into manual dependency handling, for instance with the <code>WithDepNode</code> type. I don't have a path forward for this.</p>



<a name="196353733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/196353733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#196353733">(May 05 2020 at 19:20)</a>:</h4>
<p>Hmm, so, salsa's solution here is that the dep-kind winds up getting generated at the very last crate (e.g., the driver), but very little code needs to interact with it anyhow.</p>



<a name="196353787"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/196353787" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#196353787">(May 05 2020 at 19:20)</a>:</h4>
<p>We use a trait instead of providers, so that each crate can get a <code>&amp;dyn Database</code> with all the query methods -- this has some pros and some cons</p>



<a name="196369838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/196369838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#196369838">(May 05 2020 at 21:21)</a>:</h4>
<p>The problem is not so much of the DepKind being used, but of the DepKind being required to instantiate the DepGraph. And the DepGraph is accessed directly. The first step would be to get rid of these direct accesses to the DepGraph, and replace them with some (query-based?) abstraction.</p>



<a name="196370678"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/196370678" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#196370678">(May 05 2020 at 21:28)</a>:</h4>
<p>The largest use of the DepGraph in rustc_middle is in traits/select.rs. A first step may be to integrate the selection and evaluation caches into the query system. What are the constraints on the design for these caches?</p>



<a name="204333894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204333894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Charles Lew <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204333894">(Jul 19 2020 at 05:22)</a>:</h4>
<p>I think i have an idea about this.</p>
<p>The goal: to decouple the compiler crates and remove unnecessary dependency.</p>
<p>The plan:</p>
<p>Split <code>Providers</code> into many different smaller Providers,  and store them in a <a href="https://docs.rs/typemap/0.3.3/typemap/struct.TypeMap.html"><code>TypeMap</code></a>. </p>
<p>Instead of <code>rustc_interface</code> invoking each <code>provide</code> fn separately, each module declare them into a <a href="https://docs.rs/linkme/0.2.2/linkme/"><code>distributed_slice</code></a>, and <code>rustc_interface</code> will execute these functions to populate the <code>TypeMap</code> with the provider.</p>
<p>When one crate want to execute a provider defined within another crate, it imports the definition of that provider from that crate, and retrieve an instance of it from the <code>TypeMap</code>, since everything is already initialized it can succeed in doing so.</p>
<p>In this way, i think we can decouple all the modules. No one needs to use or import a provider it neither provides nor uses.</p>



<a name="204333937"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204333937" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Charles Lew <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204333937">(Jul 19 2020 at 05:24)</a>:</h4>
<p>-</p>



<a name="204604482"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204604482" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204604482">(Jul 21 2020 at 21:23)</a>:</h4>
<p><span class="user-mention" data-user-id="116458">@Charles Lew</span> that's far too dynamic for my liking. not to mention inefficient by default (<code>TypeMap</code>, that is)</p>



<a name="204604505"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204604505" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204604505">(Jul 21 2020 at 21:23)</a>:</h4>
<blockquote>
<p>A library for safe cross-platform linker shenanigans.</p>
</blockquote>
<p>"Press X to doubt" comes to mind</p>



<a name="204604597"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204604597" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204604597">(Jul 21 2020 at 21:24)</a>:</h4>
<p>like that's top 10 famous last words of all time right there</p>



<a name="204604689"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204604689" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204604689">(Jul 21 2020 at 21:25)</a>:</h4>
<p><span class="user-mention" data-user-id="116458">@Charles Lew</span> also this just doesn't seem necessary? look at how salsa does it. and even salsa seems a bit too much for rustc for me (but arguably doable)</p>



<a name="204604975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204604975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204604975">(Jul 21 2020 at 21:29)</a>:</h4>
<blockquote>
<p>The largest use of the DepGraph in rustc_middle is in traits/select.rs. A first step may be to integrate the selection and evaluation caches into the query system.</p>
</blockquote>
<p><span class="user-mention" data-user-id="248906">@cjgillot</span> sorry, I just saw this - couldn't we just make that code be in a crate that depends on all the crates that <em>define</em> queries? (not necessary all the crates that <em>implement</em> them)</p>



<a name="204605159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/233931-t-compiler/major%20changes/topic/Decentralize%20queries%20compiler-team%23277/near/204605159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> eddyb <a href="https://rust-lang.github.io/zulip_archive/stream/233931-t-compiler/major-changes/topic/Decentralize.20queries.20compiler-team.23277.html#204605159">(Jul 21 2020 at 21:30)</a>:</h4>
<p><span class="user-mention" data-user-id="248906">@cjgillot</span> the problem with the trait system not being querifiable is mostly to do with cycles, which the trait system handles in a sound way, but queries in general cannot soundly depend on themselves. there also used to be inference too but we got around that with canonicalization. not sure how Chalk would integrate with incremental tbh</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>